Installation¶

Just pip install:

pip install omegaconf

If you want to try this notebook after checking out the repository be sure to run python setup.py develop at the repository root before running this code.

Creating OmegaConf objects¶

Empty¶

In [1]:
xxxxxxxxxx
3
 
1
from omegaconf import OmegaConf
2
conf = OmegaConf.create()
3
print(conf)
Outputs unchanged
{}
application/vnd.jupyter.stdout

From a dictionary¶

In [2]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.create(dict(k='v',list=[1,dict(a='1',b='2')]))
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
k: v
list:
- 1
- a: '1'
  b: '2'
application/vnd.jupyter.stdout

From a list¶

In [3]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.create([1, dict(a=10, b=dict(a=10))])
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
- 1
- a: 10
  b:
    a: 10
application/vnd.jupyter.stdout

From a yaml file¶

In [4]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/example.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  port: 80
log:
  file: ???
  rotation: 3600
users:
- user1
- user2
application/vnd.jupyter.stdout

From a yaml string¶

In [5]:
xxxxxxxxxx
9
 
1
yaml = """
2
a: b
3
b: c
4
list:
5
- item1
6
- item2
7
"""
8
conf = OmegaConf.create(yaml)
9
print(OmegaConf.to_yaml(conf))
Outputs unchanged
a: b
b: c
list:
- item1
- item2
application/vnd.jupyter.stdout

From a dot-list¶

In [6]:
xxxxxxxxxx
3
 
1
dot_list = ["a.aa.aaa=1", "a.aa.bbb=2", "a.bb.aaa=3", "a.bb.bbb=4"]
2
conf = OmegaConf.from_dotlist(dot_list)
3
print(OmegaConf.to_yaml(conf))
Outputs unchanged
a:
  aa:
    aaa: 1
    bbb: 2
  bb:
    aaa: 3
    bbb: 4
application/vnd.jupyter.stdout

From command line arguments¶

To parse the content of sys.arg:

In [7]:
xxxxxxxxxx
5
 
1
# Simulating command line arguments
2
import sys
3
sys.argv = ['your-program.py', 'server.port=82', 'log.file=log2.txt']
4
conf = OmegaConf.from_cli()
5
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  port: 82
log:
  file: log2.txt
application/vnd.jupyter.stdout

Access and manipulation¶

Input yaml file:

In [8]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/example.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  port: 80
log:
  file: ???
  rotation: 3600
users:
- user1
- user2
application/vnd.jupyter.stdout

Object style access:¶

In [9]:
xxxxxxxxxx
1
 
1
conf.server.port
Outputs unchanged
80
text/plain

dictionary style access¶

In [10]:
xxxxxxxxxx
1
 
1
conf['log']['rotation']
Outputs unchanged
3600
text/plain

items in list¶

In [11]:
xxxxxxxxxx
1
 
1
conf.users[0]
Outputs unchanged
'user1'
text/plain

Changing existing keys¶

In [12]:
xxxxxxxxxx
1
 
1
conf.server.port = 81

Adding new keys¶

In [13]:
xxxxxxxxxx
1
 
1
conf.server.hostname = "localhost"

Adding a new dictionary¶

In [14]:
xxxxxxxxxx
1
 
1
conf.database = {'hostname': 'database01', 'port': 3306}

providing default values¶

In [15]:
xxxxxxxxxx
2
 
1
conf.missing_key or 'a default value'
2
conf.get('missing_key', 'a default value')
Outputs unchanged
'a default value'
text/plain

Accessing mandatory values¶

Accessing fields with the value ??? will cause a MissingMandatoryValue exception. Use this to indicate that the value must be set before accessing.

In [16]:
xxxxxxxxxx
5
 
1
import pytest
2
from omegaconf import MissingMandatoryValue
3
​
4
with pytest.raises(MissingMandatoryValue):
5
    conf.log.file

Variable interpolation¶

OmegaConf support variable interpolation, Interpolations are evaluated lazily on access.

Config node interpolation¶

Interpolations are evaluated lazily on field access.
Note below that when printed the interpolations are not resolved.
They get resolved once you access them.

In [17]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/config_interpolation.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  host: localhost
  port: 80
client:
  url: http://${server.host}:${server.port}/
  server_port: ${server.port}
application/vnd.jupyter.stdout
In [18]:
xxxxxxxxxx
4
 
1
# Primitive interpolation types are inherited from the referenced value
2
print("conf.client.server_port: ", conf.client.server_port, type(conf.client.server_port).__name__)
3
# Composite interpolation types are always string
4
print("conf.client.url: ", conf.client.url, type(conf.client.url).__name__)
Outputs unchanged
conf.client.server_port:  80 int
conf.client.url:  http://localhost:80/ str
application/vnd.jupyter.stdout

to_yaml() will resolve interpolations if resolve=True is passed

In [19]:
xxxxxxxxxx
1
 
1
print(OmegaConf.to_yaml(conf, resolve=True))
Outputs unchanged
server:
  host: localhost
  port: 80
client:
  url: http://localhost:80/
  server_port: 80
application/vnd.jupyter.stdout

Interpolations may be nested, enabling more advanced behavior like dynamically selecting a sub-config:

In [20]:
In [20]:
xxxxxxxxxx
10
 
1
cfg = OmegaConf.create("""
2
plans:
3
    A: plan A
4
    B: plan B
5
selected_plan: A
6
plan: ${plans.${selected_plan}}
7
""")
8
print(f"Default: cfg.plan = {cfg.plan}")
9
cfg.selected_plan = "B"
10
print(f"After selecting plan B: cfg.plan = {cfg.plan}")
⇛⇚
xxxxxxxxxx
15
 
1
from textwrap import dedent
2
cfg = OmegaConf.create(
3
    dedent(
4
        """\
5
        plans:
6
            A: plan A
7
            B: plan B
8
        selected_plan: A
9
        plan: ${plans.${selected_plan}}
10
        """
11
    )
12
)
13
print(f"Default: cfg.plan = {cfg.plan}")
14
cfg.selected_plan = "B"
15
print(f"After selecting plan B: cfg.plan = {cfg.plan}")
Outputs unchanged
Default: cfg.plan = plan A
After selecting plan B: cfg.plan = plan B
application/vnd.jupyter.stdout

Environment variable interpolation¶

Environment variable interpolation is also supported.

In [21]:
xxxxxxxxxx
3
 
1
# Let's set up the environment first (only needed for this demonstration)
2
import os
3
os.environ['USER'] = 'omry'

Here is an example config file interpolates with the USER environment variable:

In [22]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/env_interpolation.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs unchanged
user:
  name: ${env:USER}
  home: /home/${env:USER}
application/vnd.jupyter.stdout
In [23]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/env_interpolation.yaml')
2
print(OmegaConf.to_yaml(conf, resolve=True))
Outputs unchanged
user:
  name: omry
  home: /home/omry
application/vnd.jupyter.stdout

You can specify a default value to use in case the environment variable is not defined. The following example sets abc123 as the the default value when DB_PASSWORD is not defined.

In [24]:
In [24]:
xxxxxxxxxx
6
 
1
os.environ.pop('DB_PASSWORD', None)  # ensure env variable does not exist
2
cfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}'}})
3
print(repr(cfg.database.password))
4
OmegaConf.clear_cache(cfg) # clear resolver cache
5
os.environ["DB_PASSWORD"] = 'secret'
6
print(repr(cfg.database.password))
⇛⇚
xxxxxxxxxx
3
 
1
os.environ.pop('DB_PASSWORD', None)  # ensure env variable does not exist
2
cfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}'}})
3
print(repr(cfg.database.password))
Outputs changed
Output added
'abc123'
application/vnd.jupyter.stdout
Output deleted
'abc123'
'secret'
application/vnd.jupyter.stdout

Environment variables are parsed when they are recognized as valid quantities that may be evaluated (e.g., int, float, dict, list):

In [25]:
In [25]:
xxxxxxxxxx
10
 
1
cfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}',
2
                                    'user': 'someuser',
3
                                    'port': '${env:DB_PORT,3306}',
4
                                    'nodes': '${env:DB_NODES,[]}'}})
5
os.environ["DB_PORT"] = '3308'
6
print(repr(cfg.database.port))  # converted to int
7
os.environ["DB_NODES"] = '[host1, host2, host3]'
8
print(repr(cfg.database.nodes))  # converted to list
9
os.environ["DB_PASSWORD"] = 'a%#@~{}$*&^?/<'
10
print(repr(cfg.database.password))  # kept as a string
⇛⇚
xxxxxxxxxx
12
 
1
cfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}',
2
                                    'user': 'someuser',
3
                                    'port': '${env:DB_PORT,3306}',
4
                                    'nodes': '${env:DB_NODES,[]}'}})
5
​
6
os.environ["DB_PORT"] = '3308'  # integer
7
os.environ["DB_NODES"] = '[host1, host2, host3]'  # list
8
os.environ["DB_PASSWORD"] = 'a%#@~{}$*&^?/<'  # string
9
​
10
print(repr(cfg.database.port))
11
print(repr(cfg.database.nodes))
12
print(repr(cfg.database.password))
Outputs unchanged
3308
['host1', 'host2', 'host3']
'a%#@~{}$*&^?/<'
application/vnd.jupyter.stdout

Custom interpolations¶

You can add additional interpolation types using custom resolvers. The example below creates a resolver that adds 10 to the given value.

In [26]:
xxxxxxxxxx
3
 
1
OmegaConf.new_register_resolver("plus_10", lambda x: x + 10)
2
conf = OmegaConf.create({'key': '${plus_10:990}'})
3
conf.key
Outputs unchanged
1000
text/plain

You can take advantage of nested interpolations to perform custom operations over variables:

In [27]:
xxxxxxxxxx
3
 
1
OmegaConf.new_register_resolver("plus", lambda x, y: x + y)
2
conf = OmegaConf.create({"a": 1, "b": 2, "a_plus_b": "${plus:${a},${b}}"})
3
conf.a_plus_b
Outputs unchanged
3
text/plain

By default a custom resolver’s output is cached, so that when it is called with the same inputs we always return the same value. This behavior may be disabled by setting use_cache=False:

In [28]:
xxxxxxxxxx
15
 
1
import random
2
random.seed(1234)
3
​
4
OmegaConf.new_register_resolver("cached", random.random)
5
OmegaConf.new_register_resolver("uncached", random.random, use_cache=False)
6
​
7
cfg = OmegaConf.create({"cached": "${cached:}", "uncached": "${uncached:}"})
8
​
9
print("With cache: ")
10
print(cfg.cached)
11
print(cfg.cached)  # same as above
12
​
13
print("Without cache: ")
14
print(cfg.uncached)
15
print(cfg.uncached)  # *not* the same as above
Metadata changed
xxxxxxxxxx
With cache: 
0.9664535356921388
0.9664535356921388
Without cache: 
0.4407325991753527
0.007491470058587191
application/vnd.jupyter.stdout
In [28]:
xxxxxxxxxx
18
 
1
import random
2
random.seed(1234)
3
​
4
OmegaConf.new_register_resolver("randint",
5
                                lambda a, b: random.randint(a, b))
6
c = OmegaConf.create({"x": "${randint:0, 1000}"})
7
print("With cache:")
8
print(f"c.x = {repr(c.x)}")
9
print(f"c.x = {repr(c.x)}")  # same as above thanks to the cache
10
​
11
random.seed(1234)
12
OmegaConf.new_register_resolver("randint_nocache",
13
                                lambda a, b: random.randint(a, b),
14
                                use_cache=False)  # <== disable cache!
15
c = OmegaConf.create({"x": "${randint_nocache:0, 1000}"})
16
print("Without cache:")
17
print(f"c.x = {repr(c.x)}")
18
print(f"c.x = {repr(c.x)}")  # not the same anymore since the cache is disabled
Metadata changed
xxxxxxxxxx
With cache:
c.x = 989
c.x = 989
Without cache:
c.x = 989
c.x = 796
application/vnd.jupyter.stdout

Merging configurations¶

Merging configurations enables the creation of reusable configuration files for each logical component instead of a single config file for each variation of your task.

Machine learning experiment example:

conf = OmegaConf.merge(base_cfg, model_cfg, optimizer_cfg, dataset_cfg)

Web server configuration example:

conf = OmegaConf.merge(server_cfg, plugin1_cfg, site1_cfg, site2_cfg)

The following example creates two configs from files, and one from the cli. It then combines them into a single object. Note how the port changes to 82, and how the users lists are combined.

In [29]:
xxxxxxxxxx
2
 
1
base_conf = OmegaConf.load('../source/example2.yaml')
2
print(OmegaConf.to_yaml(base_conf))
Outputs unchanged
server:
  port: 80
users:
- user1
- user2
application/vnd.jupyter.stdout
In [30]:
xxxxxxxxxx
2
 
1
second_conf = OmegaConf.load('../source/example3.yaml')
2
print(OmegaConf.to_yaml(second_conf))
Outputs unchanged
log:
  file: log.txt
application/vnd.jupyter.stdout
In [31]:
xxxxxxxxxx
11
 
1
from omegaconf import OmegaConf
2
import sys
3
​
4
# Merge configs:
5
conf = OmegaConf.merge(base_conf, second_conf)
6
​
7
# Simulate command line arguments
8
sys.argv = ['program.py', 'server.port=82']
9
# Merge with cli arguments
10
conf.merge_with_cli()
11
print(OmegaConf.to_yaml(conf))
Outputs unchanged
server:
  port: 82
users:
- user1
- user2
log:
  file: log.txt
application/vnd.jupyter.stdout